qrep: derive pull columns from source instead of catalog schema - #4583
Conversation
The QRep pull path loads the table schema from table_schema_mapping when Exclude columns or child table ranges are set, but the row was only written when SetupWatermarkTableOnDestination was true. Standalone QRep mirrors to non-PG destinations with excluded columns hit "no rows in result set". Move setupTableSchema out from behind the SetupWatermarkTableOnDestination guard so the schema is always available to the pull path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Only standalone mirrors (ParentMirrorName == FlowJobName) lack a CDC setup_flow to populate table_schema_mapping; CDC initial-load children already have it under ParentMirrorName. Guard the unconditional setupTableSchema so CDC children don't run a redundant GetTableSchema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
The QRep pull path built the SELECT column list (for Exclude / child table ranges) by loading the table schema from table_schema_mapping keyed by (ParentMirrorName, DestinationTableIdentifier). That row can be absent for CDC snapshots after resync/table-addition (the destination is renamed with a _resync suffix), producing "failed to load table schema: no rows in result set". Use GetSelectedColumns to read the column list directly from the source's pg_attribute, matching how getTableSchemaForTable already builds it. This removes the catalog dependency and fixes standalone, CDC, resync and table-addition paths alike. catalogPool is no longer needed by corePullQRepRecords. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Removes QRep pull-time dependence on table_schema_mapping by deriving the SELECT column list directly from the source table (via pg_attribute), preventing pulls from hard-failing when the catalog schema row is missing.
Changes:
- Stop loading table schema from the catalog for building the
SELECTcolumn list during QRep pulls. - Derive selected columns from the source table using
GetSelectedColumnswhen excludes or child-table range partitions are involved. - Drop
catalogPoolusage fromcorePullQRepRecords(callers keep the parameter but ignore it).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| AND c.relname = $2 | ||
| AND a.attnum > 0 | ||
| AND NOT a.attisdropped ` + excludedColumnsSQL | ||
| AND NOT a.attisdropped` + excludedColumnsSQL + ` ORDER BY a.attnum` |
There was a problem hiding this comment.
optional, suggested by copilot to keep columns order consistent across calls
ilidemi
left a comment
There was a problem hiding this comment.
Would prefer to first understand what led to the error before making a change just in case. What if there's a consistency issue
I explained it in internal discussion, posting here as well: |
ilidemi
left a comment
There was a problem hiding this comment.
Ah right, I missed that thread. Conceptually having a single source of truth is nicer, and waiting on full Temporal cancellation before starting resync would be a cleaner fix (if possible), but this is also bringing us closer to not needing an overarching snapshot tx and still supporting schema changes so ok enough.
### Error (sanitized) Postgres snapshot partitions fail during `qrep` pull with a claim that the source table has no columns: ``` [qrep] failed to pull records: table "public"."ferly_inoculum" doesn't have queriable columns ``` The line emits `errorClass=OTHER` / `errorCode=UNKNOWN` with source `other`. It has appeared on several pipes across multiple regions over the past few weeks, always in a short burst that clears within minutes. ### Investigation This a bug in PeerDB. `GetSelectedColumns` iterates `pg_attribute` rows but never checks `rows.Err()`, so when the source connection dies after the query is sent, `Next` stops immediately and the function returns an empty column list with a nil error. `corePullQRepRecords` reads that empty list as "this table has no columns" and reports it, hiding the connection failure that actually happened. - [pgx `Conn.Query`](https://pkg.go.dev/github.com/jackc/pgx/v5#Conn.Query) — documents that `Query` returns before reading any rows, so a failure after the query is sent surfaces only through `Rows.Err()`. - [PeerDB #4583](#4583) — added this pull path and its empty-column check; the message cannot predate it, and the first occurrences follow it. - [Postgres error codes](https://www.postgresql.org/docs/current/errcodes-appendix.html) — `57P01` and `57P03` cover the backend terminations that the logs show at the same second as each burst. Every occurrence in the logs lands in the same second as the source connection dropping: a source restart refusing new connections, an administrator terminating the backend, or an SSH tunnel closing. Partitions for the same table succeed before and after each burst, which rules out a genuine all-columns-excluded configuration. Confidence is high: the pgx contract explains why the empty list appears, and the logs confirm the trigger in each case. Nobody needs to act on the message itself — the fix removes it, and the real connection error takes its place. Two things worth a reviewer's attention: a permanently misconfigured exclusion list would still produce the same message legitimately, and that case belongs in validation rather than here; and `SSH Tunnel Closed: EOF` is matched by the `io.EOF` branch before it reaches the SSH-tunnel branch, which predates this change. ### Change - `GetSelectedColumns` now checks `rows.Err()` after the iteration loop and wraps it as `error getting selected columns for table %s`, matching the check its sibling function in the same file already performs. - All three callers — the `qrep` pull, table-schema fetch, and validation — previously read a truncated or empty list as fact; they now receive the underlying error. - The connection failures that caused these bursts already classify on their own: `57P03` and a closed tunnel as `NOTIFY_CONNECTIVITY`, `57P01` as `NOTIFY_TERMINATE`, a bare `unexpected EOF` as `IGNORE_EOF`. No classifier change is needed. ### Verification - `TestGetSelectedColumnsReportsMidStreamConnectionLoss` breaks a live connection after the query is sent and asserts that `GetSelectedColumns` returns an error rather than an empty column list. It fails against the unfixed code with a nil error and zero columns, reproducing the production symptom exactly. - The full classifier suite is green, as is the Postgres connector package. - An internal check rebuilt each observed production failure with the post-fix wrap chain and confirmed all of them classify into an existing class; none fall through to `OTHER`. Resolves DBI-979
Problem
QRep pulls with excluded columns or child-table range partitions fail with:
The pull path (qrep_source.go) built the
SELECTcolumn list by loading the table schema fromtable_schema_mapping, keyed by(ParentMirrorName, DestinationTableIdentifier). When no row exists for that key,QueryRow().Scan()returnspgx.ErrNoRows("no rows in result set") and the pull hard-fails.This has been observed on a CDC mirror. The in-repo setup paths normally populate that row under a matching key, so the exact reason the row was absent for this mirror isn't pinned down (candidates: version skew / older mirror, or attaching to a pre-existing mirror). Regardless, the pull shouldn't depend on the catalog row being present when it can derive the columns directly from the source.
Fix
Derive the column list directly from the source via
GetSelectedColumns(which readspg_attribute) — the same primitivegetTableSchemaForTablealready uses to build the identical list, so column selection and ordering are unchanged.This removes the
table_schema_mappingdependency from the pull path entirely.catalogPoolis no longer needed bycorePullQRepRecords.🤖 Generated with Claude Code